{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "vscode": {
     "languageId": "markdown"
    }
   },
   "source": [
    "# Adaptive Retrieval for Enhanced RAG Systems\n",
    "\n",
    "In this notebook, I implement an Adaptive Retrieval system that dynamically selects the most appropriate retrieval strategy based on the type of query. This approach significantly enhances our RAG system's ability to provide accurate and relevant responses across a diverse range of questions.\n",
    "\n",
    "Different questions demand different retrieval strategies. Our system:\n",
    "\n",
    "1. Classifies the query type (Factual, Analytical, Opinion, or Contextual)\n",
    "2. Selects the appropriate retrieval strategy\n",
    "3. Executes specialized retrieval techniques\n",
    "4. Generates a tailored response"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setting Up the Environment\n",
    "We begin by importing necessary libraries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import numpy as np\n",
    "import json\n",
    "import fitz\n",
    "from openai import OpenAI\n",
    "import re"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Extracting Text from a PDF File\n",
    "To implement RAG, we first need a source of textual data. In this case, we extract text from a PDF file using the PyMuPDF library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def extract_text_from_pdf(pdf_path):\n",
    "    \"\"\"\n",
    "    Extracts text from a PDF file and prints the first `num_chars` characters.\n",
    "\n",
    "    Args:\n",
    "    pdf_path (str): Path to the PDF file.\n",
    "\n",
    "    Returns:\n",
    "    str: Extracted text from the PDF.\n",
    "    \"\"\"\n",
    "    # Open the PDF file\n",
    "    mypdf = fitz.open(pdf_path)\n",
    "    all_text = \"\"  # Initialize an empty string to store the extracted text\n",
    "\n",
    "    # Iterate through each page in the PDF\n",
    "    for page_num in range(mypdf.page_count):\n",
    "        page = mypdf[page_num]  # Get the page\n",
    "        text = page.get_text(\"text\")  # Extract text from the page\n",
    "        all_text += text  # Append the extracted text to the all_text string\n",
    "\n",
    "    return all_text  # Return the extracted text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Chunking the Extracted Text\n",
    "Once we have the extracted text, we divide it into smaller, overlapping chunks to improve retrieval accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def chunk_text(text, n, overlap):\n",
    "    \"\"\"\n",
    "    Chunks the given text into segments of n characters with overlap.\n",
    "\n",
    "    Args:\n",
    "    text (str): The text to be chunked.\n",
    "    n (int): The number of characters in each chunk.\n",
    "    overlap (int): The number of overlapping characters between chunks.\n",
    "\n",
    "    Returns:\n",
    "    List[str]: A list of text chunks.\n",
    "    \"\"\"\n",
    "    chunks = []  # Initialize an empty list to store the chunks\n",
    "    \n",
    "    # Loop through the text with a step size of (n - overlap)\n",
    "    for i in range(0, len(text), n - overlap):\n",
    "        # Append a chunk of text from index i to i + n to the chunks list\n",
    "        chunks.append(text[i:i + n])\n",
    "\n",
    "    return chunks  # Return the list of text chunks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setting Up the OpenAI API Client\n",
    "We initialize the OpenAI client to generate embeddings and responses."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize the OpenAI client with the base URL and API key\n",
    "client = OpenAI(\n",
    "    base_url=\"http://localhost:11434/v1/\",\n",
    "    api_key=\"ollama\"  # Ollama doesn't require a real API key, but the client needs a value\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Simple Vector Store Implementation\n",
    "We'll create a basic vector store to manage document chunks and their embeddings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "class SimpleVectorStore:\n",
    "    \"\"\"\n",
    "    A simple vector store implementation using NumPy.\n",
    "    \"\"\"\n",
    "    def __init__(self):\n",
    "        \"\"\"\n",
    "        Initialize the vector store.\n",
    "        \"\"\"\n",
    "        self.vectors = []  # List to store embedding vectors\n",
    "        self.texts = []  # List to store original texts\n",
    "        self.metadata = []  # List to store metadata for each text\n",
    "    \n",
    "    def add_item(self, text, embedding, metadata=None):\n",
    "        \"\"\"\n",
    "        Add an item to the vector store.\n",
    "\n",
    "        Args:\n",
    "        text (str): The original text.\n",
    "        embedding (List[float]): The embedding vector.\n",
    "        metadata (dict, optional): Additional metadata.\n",
    "        \"\"\"\n",
    "        self.vectors.append(np.array(embedding))  # Convert embedding to numpy array and add to vectors list\n",
    "        self.texts.append(text)  # Add the original text to texts list\n",
    "        self.metadata.append(metadata or {})  # Add metadata to metadata list, default to empty dict if None\n",
    "    \n",
    "    def similarity_search(self, query_embedding, k=5, filter_func=None):\n",
    "        \"\"\"\n",
    "        Find the most similar items to a query embedding.\n",
    "\n",
    "        Args:\n",
    "        query_embedding (List[float]): Query embedding vector.\n",
    "        k (int): Number of results to return.\n",
    "        filter_func (callable, optional): Function to filter results.\n",
    "\n",
    "        Returns:\n",
    "        List[Dict]: Top k most similar items with their texts and metadata.\n",
    "        \"\"\"\n",
    "        if not self.vectors:\n",
    "            return []  # Return empty list if no vectors are stored\n",
    "        \n",
    "        # Convert query embedding to numpy array\n",
    "        query_vector = np.array(query_embedding)\n",
    "        \n",
    "        # Calculate similarities using cosine similarity\n",
    "        similarities = []\n",
    "        for i, vector in enumerate(self.vectors):\n",
    "            # Apply filter if provided\n",
    "            if filter_func and not filter_func(self.metadata[i]):\n",
    "                continue\n",
    "                \n",
    "            # Calculate cosine similarity\n",
    "            similarity = np.dot(query_vector, vector) / (np.linalg.norm(query_vector) * np.linalg.norm(vector))\n",
    "            similarities.append((i, similarity))  # Append index and similarity score\n",
    "        \n",
    "        # Sort by similarity (descending)\n",
    "        similarities.sort(key=lambda x: x[1], reverse=True)\n",
    "        \n",
    "        # Return top k results\n",
    "        results = []\n",
    "        for i in range(min(k, len(similarities))):\n",
    "            idx, score = similarities[i]\n",
    "            results.append({\n",
    "                \"text\": self.texts[idx],  # Add the text\n",
    "                \"metadata\": self.metadata[idx],  # Add the metadata\n",
    "                \"similarity\": score  # Add the similarity score\n",
    "            })\n",
    "        \n",
    "        return results  # Return the list of top k results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating Embeddings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_embeddings(text, model=\"bge-m3:latest\"):\n",
    "    \"\"\"\n",
    "    Creates embeddings for the given text.\n",
    "\n",
    "    Args:\n",
    "    text (str or List[str]): The input text(s) for which embeddings are to be created.\n",
    "    model (str): The model to be used for creating embeddings.\n",
    "\n",
    "    Returns:\n",
    "    List[float] or List[List[float]]: The embedding vector(s).\n",
    "    \"\"\"\n",
    "    # Handle both string and list inputs by converting string input to a list\n",
    "    input_text = text if isinstance(text, list) else [text]\n",
    "    \n",
    "    # Create embeddings for the input text using the specified model\n",
    "    response = client.embeddings.create(\n",
    "        model=model,\n",
    "        input=input_text\n",
    "    )\n",
    "    \n",
    "    # If the input was a single string, return just the first embedding\n",
    "    if isinstance(text, str):\n",
    "        return response.data[0].embedding\n",
    "    \n",
    "    # Otherwise, return all embeddings for the list of texts\n",
    "    return [item.embedding for item in response.data]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Document Processing Pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def process_document(pdf_path, chunk_size=1000, chunk_overlap=200):\n",
    "    \"\"\"\n",
    "    Process a document for use with adaptive retrieval.\n",
    "\n",
    "    Args:\n",
    "    pdf_path (str): Path to the PDF file.\n",
    "    chunk_size (int): Size of each chunk in characters.\n",
    "    chunk_overlap (int): Overlap between chunks in characters.\n",
    "\n",
    "    Returns:\n",
    "    Tuple[List[str], SimpleVectorStore]: Document chunks and vector store.\n",
    "    \"\"\"\n",
    "    # Extract text from the PDF file\n",
    "    print(\"Extracting text from PDF...\")\n",
    "    extracted_text = extract_text_from_pdf(pdf_path)\n",
    "    \n",
    "    # Chunk the extracted text\n",
    "    print(\"Chunking text...\")\n",
    "    chunks = chunk_text(extracted_text, chunk_size, chunk_overlap)\n",
    "    print(f\"Created {len(chunks)} text chunks\")\n",
    "    \n",
    "    # Create embeddings for the text chunks\n",
    "    print(\"Creating embeddings for chunks...\")\n",
    "    chunk_embeddings = create_embeddings(chunks)\n",
    "    \n",
    "    # Initialize the vector store\n",
    "    store = SimpleVectorStore()\n",
    "    \n",
    "    # Add each chunk and its embedding to the vector store with metadata\n",
    "    for i, (chunk, embedding) in enumerate(zip(chunks, chunk_embeddings)):\n",
    "        store.add_item(\n",
    "            text=chunk,\n",
    "            embedding=embedding,\n",
    "            metadata={\"index\": i, \"source\": pdf_path}\n",
    "        )\n",
    "    \n",
    "    print(f\"Added {len(chunks)} chunks to the vector store\")\n",
    "    \n",
    "    # Return the chunks and the vector store\n",
    "    return chunks, store"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Query Classification"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def classify_query(query, model=\"meta-llama/Llama-3.2-3B-Instruct\"):\n",
    "    \"\"\"\n",
    "    Classify a query into one of four categories: Factual, Analytical, Opinion, or Contextual.\n",
    "    \n",
    "    Args:\n",
    "        query (str): User query\n",
    "        model (str): LLM model to use\n",
    "        \n",
    "    Returns:\n",
    "        str: Query category\n",
    "    \"\"\"\n",
    "    # Define the system prompt to guide the AI's classification\n",
    "    system_prompt = \"\"\"You are an expert at classifying questions. \n",
    "        Classify the given query into exactly one of these categories:\n",
    "        - Factual: Queries seeking specific, verifiable information.\n",
    "        - Analytical: Queries requiring comprehensive analysis or explanation.\n",
    "        - Opinion: Queries about subjective matters or seeking diverse viewpoints.\n",
    "        - Contextual: Queries that depend on user-specific context.\n",
    "\n",
    "        Return ONLY the category name, without any explanation or additional text.\n",
    "    \"\"\"\n",
    "\n",
    "    # Create the user prompt with the query to be classified\n",
    "    user_prompt = f\"Classify this query: {query}\"\n",
    "    \n",
    "    # Generate the classification response from the AI model\n",
    "    response = client.chat.completions.create(\n",
    "        model=model,\n",
    "        messages=[\n",
    "            {\"role\": \"system\", \"content\": system_prompt},\n",
    "            {\"role\": \"user\", \"content\": user_prompt}\n",
    "        ],\n",
    "        temperature=0\n",
    "    )\n",
    "    \n",
    "    # Extract and strip the category from the response\n",
    "    category = response.choices[0].message.content.strip()\n",
    "    \n",
    "    # Define the list of valid categories\n",
    "    valid_categories = [\"Factual\", \"Analytical\", \"Opinion\", \"Contextual\"]\n",
    "    \n",
    "    # Ensure the returned category is valid\n",
    "    for valid in valid_categories:\n",
    "        if valid in category:\n",
    "            return valid\n",
    "    \n",
    "    # Default to \"Factual\" if classification fails\n",
    "    return \"Factual\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Implementing Specialized Retrieval Strategies\n",
    "### 1. Factual Strategy - Focus on Precision"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def factual_retrieval_strategy(query, vector_store, k=4):\n",
    "    \"\"\"\n",
    "    Retrieval strategy for factual queries focusing on precision.\n",
    "    \n",
    "    Args:\n",
    "        query (str): User query\n",
    "        vector_store (SimpleVectorStore): Vector store\n",
    "        k (int): Number of documents to return\n",
    "        \n",
    "    Returns:\n",
    "        List[Dict]: Retrieved documents\n",
    "    \"\"\"\n",
    "    print(f\"Executing Factual retrieval strategy for: '{query}'\")\n",
    "    \n",
    "    # Use LLM to enhance the query for better precision\n",
    "    system_prompt = \"\"\"You are an expert at enhancing search queries.\n",
    "        Your task is to reformulate the given factual query to make it more precise and \n",
    "        specific for information retrieval. Focus on key entities and their relationships.\n",
    "\n",
    "        Provide ONLY the enhanced query without any explanation.\n",
    "    \"\"\"\n",
    "\n",
    "    user_prompt = f\"Enhance this factual query: {query}\"\n",
    "    \n",
    "    # Generate the enhanced query using the LLM\n",
    "    response = client.chat.completions.create(\n",
    "        model=\"meta-llama/Llama-3.2-3B-Instruct\",\n",
    "        messages=[\n",
    "            {\"role\": \"system\", \"content\": system_prompt},\n",
    "            {\"role\": \"user\", \"content\": user_prompt}\n",
    "        ],\n",
    "        temperature=0\n",
    "    )\n",
    "    \n",
    "    # Extract and print the enhanced query\n",
    "    enhanced_query = response.choices[0].message.content.strip()\n",
    "    print(f\"Enhanced query: {enhanced_query}\")\n",
    "    \n",
    "    # Create embeddings for the enhanced query\n",
    "    query_embedding = create_embeddings(enhanced_query)\n",
    "    \n",
    "    # Perform initial similarity search to retrieve documents\n",
    "    initial_results = vector_store.similarity_search(query_embedding, k=k*2)\n",
    "    \n",
    "    # Initialize a list to store ranked results\n",
    "    ranked_results = []\n",
    "    \n",
    "    # Score and rank documents by relevance using LLM\n",
    "    for doc in initial_results:\n",
    "        relevance_score = score_document_relevance(enhanced_query, doc[\"text\"])\n",
    "        ranked_results.append({\n",
    "            \"text\": doc[\"text\"],\n",
    "            \"metadata\": doc[\"metadata\"],\n",
    "            \"similarity\": doc[\"similarity\"],\n",
    "            \"relevance_score\": relevance_score\n",
    "        })\n",
    "    \n",
    "    # Sort the results by relevance score in descending order\n",
    "    ranked_results.sort(key=lambda x: x[\"relevance_score\"], reverse=True)\n",
    "    \n",
    "    # Return the top k results\n",
    "    return ranked_results[:k]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Analytical Strategy - Comprehensive Coverage"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def analytical_retrieval_strategy(query, vector_store, k=4):\n",
    "    \"\"\"\n",
    "    Retrieval strategy for analytical queries focusing on comprehensive coverage.\n",
    "    \n",
    "    Args:\n",
    "        query (str): User query\n",
    "        vector_store (SimpleVectorStore): Vector store\n",
    "        k (int): Number of documents to return\n",
    "        \n",
    "    Returns:\n",
    "        List[Dict]: Retrieved documents\n",
    "    \"\"\"\n",
    "    print(f\"Executing Analytical retrieval strategy for: '{query}'\")\n",
    "    \n",
    "    # Define the system prompt to guide the AI in generating sub-questions\n",
    "    system_prompt = \"\"\"You are an expert at breaking down complex questions.\n",
    "    Generate sub-questions that explore different aspects of the main analytical query.\n",
    "    These sub-questions should cover the breadth of the topic and help retrieve \n",
    "    comprehensive information.\n",
    "\n",
    "    Return a list of exactly 3 sub-questions, one per line.\n",
    "    \"\"\"\n",
    "\n",
    "    # Create the user prompt with the main query\n",
    "    user_prompt = f\"Generate sub-questions for this analytical query: {query}\"\n",
    "    \n",
    "    # Generate the sub-questions using the LLM\n",
    "    response = client.chat.completions.create(\n",
    "        model=\"meta-llama/Llama-3.2-3B-Instruct\",\n",
    "        messages=[\n",
    "            {\"role\": \"system\", \"content\": system_prompt},\n",
    "            {\"role\": \"user\", \"content\": user_prompt}\n",
    "        ],\n",
    "        temperature=0.3\n",
    "    )\n",
    "    \n",
    "    # Extract and clean the sub-questions\n",
    "    sub_queries = response.choices[0].message.content.strip().split('\\n')\n",
    "    sub_queries = [q.strip() for q in sub_queries if q.strip()]\n",
    "    print(f\"Generated sub-queries: {sub_queries}\")\n",
    "    \n",
    "    # Retrieve documents for each sub-query\n",
    "    all_results = []\n",
    "    for sub_query in sub_queries:\n",
    "        # Create embeddings for the sub-query\n",
    "        sub_query_embedding = create_embeddings(sub_query)\n",
    "        # Perform similarity search for the sub-query\n",
    "        results = vector_store.similarity_search(sub_query_embedding, k=2)\n",
    "        all_results.extend(results)\n",
    "    \n",
    "    # Ensure diversity by selecting from different sub-query results\n",
    "    # Remove duplicates (same text content)\n",
    "    unique_texts = set()\n",
    "    diverse_results = []\n",
    "    \n",
    "    for result in all_results:\n",
    "        if result[\"text\"] not in unique_texts:\n",
    "            unique_texts.add(result[\"text\"])\n",
    "            diverse_results.append(result)\n",
    "    \n",
    "    # If we need more results to reach k, add more from initial results\n",
    "    if len(diverse_results) < k:\n",
    "        # Direct retrieval for the main query\n",
    "        main_query_embedding = create_embeddings(query)\n",
    "        main_results = vector_store.similarity_search(main_query_embedding, k=k)\n",
    "        \n",
    "        for result in main_results:\n",
    "            if result[\"text\"] not in unique_texts and len(diverse_results) < k:\n",
    "                unique_texts.add(result[\"text\"])\n",
    "                diverse_results.append(result)\n",
    "    \n",
    "    # Return the top k diverse results\n",
    "    return diverse_results[:k]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Opinion Strategy - Diverse Perspectives"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "def opinion_retrieval_strategy(query, vector_store, k=4):\n",
    "    \"\"\"\n",
    "    Retrieval strategy for opinion queries focusing on diverse perspectives.\n",
    "    \n",
    "    Args:\n",
    "        query (str): User query\n",
    "        vector_store (SimpleVectorStore): Vector store\n",
    "        k (int): Number of documents to return\n",
    "        \n",
    "    Returns:\n",
    "        List[Dict]: Retrieved documents\n",
    "    \"\"\"\n",
    "    print(f\"Executing Opinion retrieval strategy for: '{query}'\")\n",
    "    \n",
    "    # Define the system prompt to guide the AI in identifying different perspectives\n",
    "    system_prompt = \"\"\"You are an expert at identifying different perspectives on a topic.\n",
    "        For the given query about opinions or viewpoints, identify different perspectives \n",
    "        that people might have on this topic.\n",
    "\n",
    "        Return a list of exactly 3 different viewpoint angles, one per line.\n",
    "    \"\"\"\n",
    "\n",
    "    # Create the user prompt with the main query\n",
    "    user_prompt = f\"Identify different perspectives on: {query}\"\n",
    "    \n",
    "    # Generate the different perspectives using the LLM\n",
    "    response = client.chat.completions.create(\n",
    "        model=\"meta-llama/Llama-3.2-3B-Instruct\",\n",
    "        messages=[\n",
    "            {\"role\": \"system\", \"content\": system_prompt},\n",
    "            {\"role\": \"user\", \"content\": user_prompt}\n",
    "        ],\n",
    "        temperature=0.3\n",
    "    )\n",
    "    \n",
    "    # Extract and clean the viewpoints\n",
    "    viewpoints = response.choices[0].message.content.strip().split('\\n')\n",
    "    viewpoints = [v.strip() for v in viewpoints if v.strip()]\n",
    "    print(f\"Identified viewpoints: {viewpoints}\")\n",
    "    \n",
    "    # Retrieve documents representing each viewpoint\n",
    "    all_results = []\n",
    "    for viewpoint in viewpoints:\n",
    "        # Combine the main query with the viewpoint\n",
    "        combined_query = f\"{query} {viewpoint}\"\n",
    "        # Create embeddings for the combined query\n",
    "        viewpoint_embedding = create_embeddings(combined_query)\n",
    "        # Perform similarity search for the combined query\n",
    "        results = vector_store.similarity_search(viewpoint_embedding, k=2)\n",
    "        \n",
    "        # Mark results with the viewpoint they represent\n",
    "        for result in results:\n",
    "            result[\"viewpoint\"] = viewpoint\n",
    "        \n",
    "        # Add the results to the list of all results\n",
    "        all_results.extend(results)\n",
    "    \n",
    "    # Select a diverse range of opinions\n",
    "    # Ensure we get at least one document from each viewpoint if possible\n",
    "    selected_results = []\n",
    "    for viewpoint in viewpoints:\n",
    "        # Filter documents by viewpoint\n",
    "        viewpoint_docs = [r for r in all_results if r.get(\"viewpoint\") == viewpoint]\n",
    "        if viewpoint_docs:\n",
    "            selected_results.append(viewpoint_docs[0])\n",
    "    \n",
    "    # Fill remaining slots with highest similarity docs\n",
    "    remaining_slots = k - len(selected_results)\n",
    "    if remaining_slots > 0:\n",
    "        # Sort remaining docs by similarity\n",
    "        remaining_docs = [r for r in all_results if r not in selected_results]\n",
    "        remaining_docs.sort(key=lambda x: x[\"similarity\"], reverse=True)\n",
    "        selected_results.extend(remaining_docs[:remaining_slots])\n",
    "    \n",
    "    # Return the top k results\n",
    "    return selected_results[:k]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Contextual Strategy - User Context Integration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "def contextual_retrieval_strategy(query, vector_store, k=4, user_context=None):\n",
    "    \"\"\"\n",
    "    Retrieval strategy for contextual queries integrating user context.\n",
    "    \n",
    "    Args:\n",
    "        query (str): User query\n",
    "        vector_store (SimpleVectorStore): Vector store\n",
    "        k (int): Number of documents to return\n",
    "        user_context (str): Additional user context\n",
    "        \n",
    "    Returns:\n",
    "        List[Dict]: Retrieved documents\n",
    "    \"\"\"\n",
    "    print(f\"Executing Contextual retrieval strategy for: '{query}'\")\n",
    "    \n",
    "    # If no user context provided, try to infer it from the query\n",
    "    if not user_context:\n",
    "        system_prompt = \"\"\"You are an expert at understanding implied context in questions.\n",
    "For the given query, infer what contextual information might be relevant or implied \n",
    "but not explicitly stated. Focus on what background would help answering this query.\n",
    "\n",
    "Return a brief description of the implied context.\"\"\"\n",
    "\n",
    "        user_prompt = f\"Infer the implied context in this query: {query}\"\n",
    "        \n",
    "        # Generate the inferred context using the LLM\n",
    "        response = client.chat.completions.create(\n",
    "            model=\"meta-llama/Llama-3.2-3B-Instruct\",\n",
    "            messages=[\n",
    "                {\"role\": \"system\", \"content\": system_prompt},\n",
    "                {\"role\": \"user\", \"content\": user_prompt}\n",
    "            ],\n",
    "            temperature=0.1\n",
    "        )\n",
    "        \n",
    "        # Extract and print the inferred context\n",
    "        user_context = response.choices[0].message.content.strip()\n",
    "        print(f\"Inferred context: {user_context}\")\n",
    "    \n",
    "    # Reformulate the query to incorporate context\n",
    "    system_prompt = \"\"\"You are an expert at reformulating questions with context.\n",
    "    Given a query and some contextual information, create a more specific query that \n",
    "    incorporates the context to get more relevant information.\n",
    "\n",
    "    Return ONLY the reformulated query without explanation.\"\"\"\n",
    "\n",
    "    user_prompt = f\"\"\"\n",
    "    Query: {query}\n",
    "    Context: {user_context}\n",
    "\n",
    "    Reformulate the query to incorporate this context:\"\"\"\n",
    "    \n",
    "    # Generate the contextualized query using the LLM\n",
    "    response = client.chat.completions.create(\n",
    "        model=\"meta-llama/Llama-3.2-3B-Instruct\",\n",
    "        messages=[\n",
    "            {\"role\": \"system\", \"content\": system_prompt},\n",
    "            {\"role\": \"user\", \"content\": user_prompt}\n",
    "        ],\n",
    "        temperature=0\n",
    "    )\n",
    "    \n",
    "    # Extract and print the contextualized query\n",
    "    contextualized_query = response.choices[0].message.content.strip()\n",
    "    print(f\"Contextualized query: {contextualized_query}\")\n",
    "    \n",
    "    # Retrieve documents based on the contextualized query\n",
    "    query_embedding = create_embeddings(contextualized_query)\n",
    "    initial_results = vector_store.similarity_search(query_embedding, k=k*2)\n",
    "    \n",
    "    # Rank documents considering both relevance and user context\n",
    "    ranked_results = []\n",
    "    \n",
    "    for doc in initial_results:\n",
    "        # Score document relevance considering the context\n",
    "        context_relevance = score_document_context_relevance(query, user_context, doc[\"text\"])\n",
    "        ranked_results.append({\n",
    "            \"text\": doc[\"text\"],\n",
    "            \"metadata\": doc[\"metadata\"],\n",
    "            \"similarity\": doc[\"similarity\"],\n",
    "            \"context_relevance\": context_relevance\n",
    "        })\n",
    "    \n",
    "    # Sort by context relevance and return top k results\n",
    "    ranked_results.sort(key=lambda x: x[\"context_relevance\"], reverse=True)\n",
    "    return ranked_results[:k]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Helper Functions for Document Scoring"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "def score_document_relevance(query, document, model=\"meta-llama/Llama-3.2-3B-Instruct\"):\n",
    "    \"\"\"\n",
    "    Score document relevance to a query using LLM.\n",
    "    \n",
    "    Args:\n",
    "        query (str): User query\n",
    "        document (str): Document text\n",
    "        model (str): LLM model\n",
    "        \n",
    "    Returns:\n",
    "        float: Relevance score from 0-10\n",
    "    \"\"\"\n",
    "    # System prompt to instruct the model on how to rate relevance\n",
    "    system_prompt = \"\"\"You are an expert at evaluating document relevance.\n",
    "        Rate the relevance of a document to a query on a scale from 0 to 10, where:\n",
    "        0 = Completely irrelevant\n",
    "        10 = Perfectly addresses the query\n",
    "\n",
    "        Return ONLY a numerical score between 0 and 10, nothing else.\n",
    "    \"\"\"\n",
    "\n",
    "    # Truncate document if it's too long\n",
    "    doc_preview = document[:1500] + \"...\" if len(document) > 1500 else document\n",
    "    \n",
    "    # User prompt containing the query and document preview\n",
    "    user_prompt = f\"\"\"\n",
    "        Query: {query}\n",
    "\n",
    "        Document: {doc_preview}\n",
    "\n",
    "        Relevance score (0-10):\n",
    "    \"\"\"\n",
    "    \n",
    "    # Generate response from the model\n",
    "    response = client.chat.completions.create(\n",
    "        model=model,\n",
    "        messages=[\n",
    "            {\"role\": \"system\", \"content\": system_prompt},\n",
    "            {\"role\": \"user\", \"content\": user_prompt}\n",
    "        ],\n",
    "        temperature=0\n",
    "    )\n",
    "    \n",
    "    # Extract the score from the model's response\n",
    "    score_text = response.choices[0].message.content.strip()\n",
    "    \n",
    "    # Extract numeric score using regex\n",
    "    match = re.search(r'(\\d+(\\.\\d+)?)', score_text)\n",
    "    if match:\n",
    "        score = float(match.group(1))\n",
    "        return min(10, max(0, score))  # Ensure score is within 0-10\n",
    "    else:\n",
    "        # Default score if extraction fails\n",
    "        return 5.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "def score_document_context_relevance(query, context, document, model=\"meta-llama/Llama-3.2-3B-Instruct\"):\n",
    "    \"\"\"\n",
    "    Score document relevance considering both query and context.\n",
    "    \n",
    "    Args:\n",
    "        query (str): User query\n",
    "        context (str): User context\n",
    "        document (str): Document text\n",
    "        model (str): LLM model\n",
    "        \n",
    "    Returns:\n",
    "        float: Relevance score from 0-10\n",
    "    \"\"\"\n",
    "    # System prompt to instruct the model on how to rate relevance considering context\n",
    "    system_prompt = \"\"\"You are an expert at evaluating document relevance considering context.\n",
    "        Rate the document on a scale from 0 to 10 based on how well it addresses the query\n",
    "        when considering the provided context, where:\n",
    "        0 = Completely irrelevant\n",
    "        10 = Perfectly addresses the query in the given context\n",
    "\n",
    "        Return ONLY a numerical score between 0 and 10, nothing else.\n",
    "    \"\"\"\n",
    "\n",
    "    # Truncate document if it's too long\n",
    "    doc_preview = document[:1500] + \"...\" if len(document) > 1500 else document\n",
    "    \n",
    "    # User prompt containing the query, context, and document preview\n",
    "    user_prompt = f\"\"\"\n",
    "    Query: {query}\n",
    "    Context: {context}\n",
    "\n",
    "    Document: {doc_preview}\n",
    "\n",
    "    Relevance score considering context (0-10):\n",
    "    \"\"\"\n",
    "    \n",
    "    # Generate response from the model\n",
    "    response = client.chat.completions.create(\n",
    "        model=model,\n",
    "        messages=[\n",
    "            {\"role\": \"system\", \"content\": system_prompt},\n",
    "            {\"role\": \"user\", \"content\": user_prompt}\n",
    "        ],\n",
    "        temperature=0\n",
    "    )\n",
    "    \n",
    "    # Extract the score from the model's response\n",
    "    score_text = response.choices[0].message.content.strip()\n",
    "    \n",
    "    # Extract numeric score using regex\n",
    "    match = re.search(r'(\\d+(\\.\\d+)?)', score_text)\n",
    "    if match:\n",
    "        score = float(match.group(1))\n",
    "        return min(10, max(0, score))  # Ensure score is within 0-10\n",
    "    else:\n",
    "        # Default score if extraction fails\n",
    "        return 5.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Core Adaptive Retriever"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "def adaptive_retrieval(query, vector_store, k=4, user_context=None):\n",
    "    \"\"\"\n",
    "    Perform adaptive retrieval by selecting and executing the appropriate strategy.\n",
    "    \n",
    "    Args:\n",
    "        query (str): User query\n",
    "        vector_store (SimpleVectorStore): Vector store\n",
    "        k (int): Number of documents to retrieve\n",
    "        user_context (str): Optional user context for contextual queries\n",
    "        \n",
    "    Returns:\n",
    "        List[Dict]: Retrieved documents\n",
    "    \"\"\"\n",
    "    # Classify the query to determine its type\n",
    "    query_type = classify_query(query)\n",
    "    print(f\"Query classified as: {query_type}\")\n",
    "    \n",
    "    # Select and execute the appropriate retrieval strategy based on the query type\n",
    "    if query_type == \"Factual\":\n",
    "        # Use the factual retrieval strategy for precise information\n",
    "        results = factual_retrieval_strategy(query, vector_store, k)\n",
    "    elif query_type == \"Analytical\":\n",
    "        # Use the analytical retrieval strategy for comprehensive coverage\n",
    "        results = analytical_retrieval_strategy(query, vector_store, k)\n",
    "    elif query_type == \"Opinion\":\n",
    "        # Use the opinion retrieval strategy for diverse perspectives\n",
    "        results = opinion_retrieval_strategy(query, vector_store, k)\n",
    "    elif query_type == \"Contextual\":\n",
    "        # Use the contextual retrieval strategy, incorporating user context\n",
    "        results = contextual_retrieval_strategy(query, vector_store, k, user_context)\n",
    "    else:\n",
    "        # Default to factual retrieval strategy if classification fails\n",
    "        results = factual_retrieval_strategy(query, vector_store, k)\n",
    "    \n",
    "    return results  # Return the retrieved documents"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Response Generation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "def generate_response(query, results, query_type, model=\"meta-llama/Llama-3.2-3B-Instruct\"):\n",
    "    \"\"\"\n",
    "    Generate a response based on query, retrieved documents, and query type.\n",
    "    \n",
    "    Args:\n",
    "        query (str): User query\n",
    "        results (List[Dict]): Retrieved documents\n",
    "        query_type (str): Type of query\n",
    "        model (str): LLM model\n",
    "        \n",
    "    Returns:\n",
    "        str: Generated response\n",
    "    \"\"\"\n",
    "    # Prepare context from retrieved documents by joining their texts with separators\n",
    "    context = \"\\n\\n---\\n\\n\".join([r[\"text\"] for r in results])\n",
    "    \n",
    "    # Create custom system prompt based on query type\n",
    "    if query_type == \"Factual\":\n",
    "        system_prompt = \"\"\"You are a helpful assistant providing factual information.\n",
    "    Answer the question based on the provided context. Focus on accuracy and precision.\n",
    "    If the context doesn't contain the information needed, acknowledge the limitations.\"\"\"\n",
    "        \n",
    "    elif query_type == \"Analytical\":\n",
    "        system_prompt = \"\"\"You are a helpful assistant providing analytical insights.\n",
    "    Based on the provided context, offer a comprehensive analysis of the topic.\n",
    "    Cover different aspects and perspectives in your explanation.\n",
    "    If the context has gaps, acknowledge them while providing the best analysis possible.\"\"\"\n",
    "        \n",
    "    elif query_type == \"Opinion\":\n",
    "        system_prompt = \"\"\"You are a helpful assistant discussing topics with multiple viewpoints.\n",
    "    Based on the provided context, present different perspectives on the topic.\n",
    "    Ensure fair representation of diverse opinions without showing bias.\n",
    "    Acknowledge where the context presents limited viewpoints.\"\"\"\n",
    "        \n",
    "    elif query_type == \"Contextual\":\n",
    "        system_prompt = \"\"\"You are a helpful assistant providing contextually relevant information.\n",
    "    Answer the question considering both the query and its context.\n",
    "    Make connections between the query context and the information in the provided documents.\n",
    "    If the context doesn't fully address the specific situation, acknowledge the limitations.\"\"\"\n",
    "        \n",
    "    else:\n",
    "        system_prompt = \"\"\"You are a helpful assistant. Answer the question based on the provided context. If you cannot answer from the context, acknowledge the limitations.\"\"\"\n",
    "    \n",
    "    # Create user prompt by combining the context and the query\n",
    "    user_prompt = f\"\"\"\n",
    "    Context:\n",
    "    {context}\n",
    "\n",
    "    Question: {query}\n",
    "\n",
    "    Please provide a helpful response based on the context.\n",
    "    \"\"\"\n",
    "    \n",
    "    # Generate response using the OpenAI client\n",
    "    response = client.chat.completions.create(\n",
    "        model=model,\n",
    "        messages=[\n",
    "            {\"role\": \"system\", \"content\": system_prompt},\n",
    "            {\"role\": \"user\", \"content\": user_prompt}\n",
    "        ],\n",
    "        temperature=0.2\n",
    "    )\n",
    "    \n",
    "    # Return the generated response content\n",
    "    return response.choices[0].message.content"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Complete RAG Pipeline with Adaptive Retrieval"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "def rag_with_adaptive_retrieval(pdf_path, query, k=4, user_context=None):\n",
    "    \"\"\"\n",
    "    Complete RAG pipeline with adaptive retrieval.\n",
    "    \n",
    "    Args:\n",
    "        pdf_path (str): Path to PDF document\n",
    "        query (str): User query\n",
    "        k (int): Number of documents to retrieve\n",
    "        user_context (str): Optional user context\n",
    "        \n",
    "    Returns:\n",
    "        Dict: Results including query, retrieved documents, query type, and response\n",
    "    \"\"\"\n",
    "    print(\"\\n=== RAG WITH ADAPTIVE RETRIEVAL ===\")\n",
    "    print(f\"Query: {query}\")\n",
    "    \n",
    "    # Process the document to extract text, chunk it, and create embeddings\n",
    "    chunks, vector_store = process_document(pdf_path)\n",
    "    \n",
    "    # Classify the query to determine its type\n",
    "    query_type = classify_query(query)\n",
    "    print(f\"Query classified as: {query_type}\")\n",
    "    \n",
    "    # Retrieve documents using the adaptive retrieval strategy based on the query type\n",
    "    retrieved_docs = adaptive_retrieval(query, vector_store, k, user_context)\n",
    "    \n",
    "    # Generate a response based on the query, retrieved documents, and query type\n",
    "    response = generate_response(query, retrieved_docs, query_type)\n",
    "    \n",
    "    # Compile the results into a dictionary\n",
    "    result = {\n",
    "        \"query\": query,\n",
    "        \"query_type\": query_type,\n",
    "        \"retrieved_documents\": retrieved_docs,\n",
    "        \"response\": response\n",
    "    }\n",
    "    \n",
    "    print(\"\\n=== RESPONSE ===\")\n",
    "    print(response)\n",
    "    \n",
    "    return result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluation Framework"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "def evaluate_adaptive_vs_standard(pdf_path, test_queries, reference_answers=None):\n",
    "    \"\"\"\n",
    "    Compare adaptive retrieval with standard retrieval on a set of test queries.\n",
    "    \n",
    "    This function processes a document, runs both standard and adaptive retrieval methods\n",
    "    on each test query, and compares their performance. If reference answers are provided,\n",
    "    it also evaluates the quality of responses against these references.\n",
    "    \n",
    "    Args:\n",
    "        pdf_path (str): Path to PDF document to be processed as the knowledge source\n",
    "        test_queries (List[str]): List of test queries to evaluate both retrieval methods\n",
    "        reference_answers (List[str], optional): Reference answers for evaluation metrics\n",
    "        \n",
    "    Returns:\n",
    "        Dict: Evaluation results containing individual query results and overall comparison\n",
    "    \"\"\"\n",
    "    print(\"=== EVALUATING ADAPTIVE VS. STANDARD RETRIEVAL ===\")\n",
    "    \n",
    "    # Process document to extract text, create chunks and build the vector store\n",
    "    chunks, vector_store = process_document(pdf_path)\n",
    "    \n",
    "    # Initialize collection for storing comparison results\n",
    "    results = []\n",
    "    \n",
    "    # Process each test query with both retrieval methods\n",
    "    for i, query in enumerate(test_queries):\n",
    "        print(f\"\\n\\nQuery {i+1}: {query}\")\n",
    "        \n",
    "        # --- Standard retrieval approach ---\n",
    "        print(\"\\n--- Standard Retrieval ---\")\n",
    "        # Create embedding for the query\n",
    "        query_embedding = create_embeddings(query)\n",
    "        # Retrieve documents using simple vector similarity\n",
    "        standard_docs = vector_store.similarity_search(query_embedding, k=4)\n",
    "        # Generate response using a generic approach\n",
    "        standard_response = generate_response(query, standard_docs, \"General\")\n",
    "        \n",
    "        # --- Adaptive retrieval approach ---\n",
    "        print(\"\\n--- Adaptive Retrieval ---\")\n",
    "        # Classify the query to determine its type (Factual, Analytical, Opinion, Contextual)\n",
    "        query_type = classify_query(query)\n",
    "        # Retrieve documents using the strategy appropriate for this query type\n",
    "        adaptive_docs = adaptive_retrieval(query, vector_store, k=4)\n",
    "        # Generate a response tailored to the query type\n",
    "        adaptive_response = generate_response(query, adaptive_docs, query_type)\n",
    "        \n",
    "        # Store complete results for this query\n",
    "        result = {\n",
    "            \"query\": query,\n",
    "            \"query_type\": query_type,\n",
    "            \"standard_retrieval\": {\n",
    "                \"documents\": standard_docs,\n",
    "                \"response\": standard_response\n",
    "            },\n",
    "            \"adaptive_retrieval\": {\n",
    "                \"documents\": adaptive_docs,\n",
    "                \"response\": adaptive_response\n",
    "            }\n",
    "        }\n",
    "        \n",
    "        # Add reference answer if available for this query\n",
    "        if reference_answers and i < len(reference_answers):\n",
    "            result[\"reference_answer\"] = reference_answers[i]\n",
    "            \n",
    "        results.append(result)\n",
    "        \n",
    "        # Display preview of both responses for quick comparison\n",
    "        print(\"\\n--- Responses ---\")\n",
    "        print(f\"Standard: {standard_response[:200]}...\")\n",
    "        print(f\"Adaptive: {adaptive_response[:200]}...\")\n",
    "    \n",
    "    # Calculate comparative metrics if reference answers are available\n",
    "    if reference_answers:\n",
    "        comparison = compare_responses(results)\n",
    "        print(\"\\n=== EVALUATION RESULTS ===\")\n",
    "        print(comparison)\n",
    "    \n",
    "    # Return the complete evaluation results\n",
    "    return {\n",
    "        \"results\": results,\n",
    "        \"comparison\": comparison if reference_answers else \"No reference answers provided for evaluation\"\n",
    "    }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compare_responses(results):\n",
    "    \"\"\"\n",
    "    Compare standard and adaptive responses against reference answers.\n",
    "    \n",
    "    Args:\n",
    "        results (List[Dict]): Results containing both types of responses\n",
    "        \n",
    "    Returns:\n",
    "        str: Comparison analysis\n",
    "    \"\"\"\n",
    "    # Define the system prompt to guide the AI in comparing responses\n",
    "    comparison_prompt = \"\"\"You are an expert evaluator of information retrieval systems.\n",
    "    Compare the standard retrieval and adaptive retrieval responses for each query.\n",
    "    Consider factors like accuracy, relevance, comprehensiveness, and alignment with the reference answer.\n",
    "    Provide a detailed analysis of the strengths and weaknesses of each approach.\"\"\"\n",
    "    \n",
    "    # Initialize the comparison text with a header\n",
    "    comparison_text = \"# Evaluation of Standard vs. Adaptive Retrieval\\n\\n\"\n",
    "    \n",
    "    # Iterate through each result to compare responses\n",
    "    for i, result in enumerate(results):\n",
    "        # Skip if there is no reference answer for the query\n",
    "        if \"reference_answer\" not in result:\n",
    "            continue\n",
    "            \n",
    "        # Add query details to the comparison text\n",
    "        comparison_text += f\"## Query {i+1}: {result['query']}\\n\"\n",
    "        comparison_text += f\"*Query Type: {result['query_type']}*\\n\\n\"\n",
    "        comparison_text += f\"**Reference Answer:**\\n{result['reference_answer']}\\n\\n\"\n",
    "        \n",
    "        # Add standard retrieval response to the comparison text\n",
    "        comparison_text += f\"**Standard Retrieval Response:**\\n{result['standard_retrieval']['response']}\\n\\n\"\n",
    "        \n",
    "        # Add adaptive retrieval response to the comparison text\n",
    "        comparison_text += f\"**Adaptive Retrieval Response:**\\n{result['adaptive_retrieval']['response']}\\n\\n\"\n",
    "        \n",
    "        # Create the user prompt for the AI to compare the responses\n",
    "        user_prompt = f\"\"\"\n",
    "        Reference Answer: {result['reference_answer']}\n",
    "        \n",
    "        Standard Retrieval Response: {result['standard_retrieval']['response']}\n",
    "        \n",
    "        Adaptive Retrieval Response: {result['adaptive_retrieval']['response']}\n",
    "        \n",
    "        Provide a detailed comparison of the two responses.\n",
    "        \"\"\"\n",
    "        \n",
    "        # Generate the comparison analysis using the OpenAI client\n",
    "        response = client.chat.completions.create(\n",
    "            model=\"meta-llama/Llama-3.2-3B-Instruct\",\n",
    "            messages=[\n",
    "                {\"role\": \"system\", \"content\": comparison_prompt},\n",
    "                {\"role\": \"user\", \"content\": user_prompt}\n",
    "            ],\n",
    "            temperature=0.2\n",
    "        )\n",
    "        \n",
    "        # Add the AI's comparison analysis to the comparison text\n",
    "        comparison_text += f\"**Comparison Analysis:**\\n{response.choices[0].message.content}\\n\\n\"\n",
    "    \n",
    "    return comparison_text  # Return the complete comparison analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluating the Adaptive Retrieval System (Customized Queries)\n",
    "\n",
    "The final step to use the adaptive RAG evaluation system is to call the evaluate_adaptive_vs_standard() function with your PDF document and test queries:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Path to your knowledge source document\n",
    "# This PDF file contains the information that the RAG system will use\n",
    "pdf_path = \"data/AI_Information.pdf\"\n",
    "\n",
    "# Define test queries covering different query types to demonstrate \n",
    "# how adaptive retrieval handles various query intentions\n",
    "test_queries = [\n",
    "    \"What is Explainable AI (XAI)?\",                                              # Factual query - seeking definition/specific information\n",
    "    # \"How do AI ethics and governance frameworks address potential societal impacts?\",  # Analytical query - requiring comprehensive analysis\n",
    "    # \"Is AI development moving too fast for proper regulation?\",                   # Opinion query - seeking diverse perspectives\n",
    "    # \"How might explainable AI help in healthcare decisions?\",                     # Contextual query - benefits from context-awareness\n",
    "]\n",
    "\n",
    "# Reference answers for more thorough evaluation\n",
    "# These can be used to objectively assess response quality against a known standard\n",
    "reference_answers = [\n",
    "    \"Explainable AI (XAI) aims to make AI systems transparent and understandable by providing clear explanations of how decisions are made. This helps users trust and effectively manage AI technologies.\",\n",
    "    # \"AI ethics and governance frameworks address potential societal impacts by establishing guidelines and principles to ensure AI systems are developed and used responsibly. These frameworks focus on fairness, accountability, transparency, and the protection of human rights to mitigate risks and promote beneficial output.5.\",\n",
    "    # \"Opinions on whether AI development is moving too fast for proper regulation vary. Some argue that rapid advancements outpace regulatory efforts, leading to potential risks and ethical concerns. Others believe that innovation should continue at its current pace, with regulations evolving alongside to address emerging challenges.\",\n",
    "    # \"Explainable AI can significantly aid healthcare decisions by providing transparent and understandable insights into AI-driven recommendations. This transparency helps healthcare professionals trust AI systems, make informed decisions, and improve patient output by understanding the rationale behind AI suggestions.\"\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=== EVALUATING ADAPTIVE VS. STANDARD RETRIEVAL ===\n",
      "Extracting text from PDF...\n",
      "Chunking text...\n",
      "Created 42 text chunks\n",
      "Creating embeddings for chunks...\n",
      "Added 42 chunks to the vector store\n",
      "\n",
      "\n",
      "Query 1: What is Explainable AI (XAI)?\n",
      "\n",
      "--- Standard Retrieval ---\n",
      "\n",
      "--- Adaptive Retrieval ---\n",
      "Query classified as: Factual\n",
      "Executing Factual retrieval strategy for: 'What is Explainable AI (XAI)?'\n",
      "Enhanced query: What are the key applications and techniques of Explainable Artificial Intelligence (XAI) in machine learning and deep learning?\n",
      "\n",
      "--- Responses ---\n",
      "Standard: Based on the provided context, Explainable AI (XAI) is a set of techniques aimed at making AI decisions more understandable, enabling users to assess their fairness and accuracy. The goal of XAI is to...\n",
      "Adaptive: Explainable AI (XAI) is a subfield of artificial intelligence (AI) that aims to make AI systems more transparent and understandable. The primary goal of XAI is to provide insights into how AI models m...\n",
      "\n",
      "=== EVALUATION RESULTS ===\n",
      "# Evaluation of Standard vs. Adaptive Retrieval\n",
      "\n",
      "## Query 1: What is Explainable AI (XAI)?\n",
      "*Query Type: Factual*\n",
      "\n",
      "**Reference Answer:**\n",
      "Explainable AI (XAI) aims to make AI systems transparent and understandable by providing clear explanations of how decisions are made. This helps users trust and effectively manage AI technologies.\n",
      "\n",
      "**Standard Retrieval Response:**\n",
      "Based on the provided context, Explainable AI (XAI) is a set of techniques aimed at making AI decisions more understandable, enabling users to assess their fairness and accuracy. The goal of XAI is to provide insights into how AI models make decisions, enhancing trust and accountability in AI systems. This involves developing methods for explaining AI decisions, which can help users understand the reliability and fairness of AI-driven outcomes.\n",
      "\n",
      "**Adaptive Retrieval Response:**\n",
      "Explainable AI (XAI) is a subfield of artificial intelligence (AI) that aims to make AI systems more transparent and understandable. The primary goal of XAI is to provide insights into how AI models make decisions, thereby enhancing trust and accountability in AI systems.\n",
      "\n",
      "XAI techniques are being developed to explain the reasoning behind AI decisions, making it possible for users to assess the reliability and fairness of AI outputs. This is particularly important in high-stakes applications such as medical diagnosis, finance, transportation, and manufacturing, where the consequences of AI errors can be severe.\n",
      "\n",
      "By making AI systems more explainable, XAI techniques can help address concerns about the potential for unintended consequences, accountability, and responsibility in AI development and deployment.\n",
      "\n",
      "**Comparison Analysis:**\n",
      "**Comparison of Standard Retrieval Response and Adaptive Retrieval Response**\n",
      "\n",
      "The two responses provide similar information about Explainable AI (XAI), but they differ in their tone, structure, and level of detail. Here's a detailed comparison of the two responses:\n",
      "\n",
      "**Accuracy and Relevance**\n",
      "\n",
      "* Both responses accurately convey the main idea of XAI, which is to make AI systems more transparent and understandable.\n",
      "* However, the Standard Retrieval Response provides more context and background information about XAI, including its goals and applications.\n",
      "* The Adaptive Retrieval Response is more concise and to the point, but it lacks the depth and detail of the Standard Retrieval Response.\n",
      "\n",
      "**Comprehensiveness**\n",
      "\n",
      "* The Standard Retrieval Response provides a more comprehensive overview of XAI, including its techniques, goals, and applications.\n",
      "* The Adaptive Retrieval Response focuses primarily on the definition and purpose of XAI, without providing much additional context or information.\n",
      "* The Standard Retrieval Response also highlights the importance of XAI in high-stakes applications, such as medical diagnosis, finance, transportation, and manufacturing.\n",
      "\n",
      "**Alignment with Reference Answer**\n",
      "\n",
      "* Both responses align with the reference answer, but the Standard Retrieval Response is more closely aligned due to its more detailed and comprehensive explanation of XAI.\n",
      "* The Adaptive Retrieval Response is more concise and to the point, but it may not fully capture the nuances and complexities of XAI.\n",
      "\n",
      "**Strengths and Weaknesses**\n",
      "\n",
      "**Standard Retrieval Response**\n",
      "\n",
      "Strengths:\n",
      "\n",
      "* Provides a more comprehensive overview of XAI\n",
      "* Offers more context and background information\n",
      "* Aligns closely with the reference answer\n",
      "\n",
      "Weaknesses:\n",
      "\n",
      "* May be too lengthy or wordy for some readers\n",
      "* Lacks the concise and to-the-point style of the Adaptive Retrieval Response\n",
      "\n",
      "**Adaptive Retrieval Response**\n",
      "\n",
      "Strengths:\n",
      "\n",
      "* Is concise and to the point\n",
      "* Provides a clear and direct definition of XAI\n",
      "* May be more suitable for readers who prefer a brief overview\n",
      "\n",
      "Weaknesses:\n",
      "\n",
      "* Lacks depth and detail\n",
      "* Fails to provide much additional context or information about XAI\n",
      "* May not fully capture the nuances and complexities of XAI\n",
      "\n",
      "**Conclusion**\n",
      "\n",
      "The Standard Retrieval Response provides a more comprehensive and detailed overview of XAI, while the Adaptive Retrieval Response is more concise and to the point. Both responses align with the reference answer, but the Standard Retrieval Response is more closely aligned due to its more detailed and comprehensive explanation of XAI.\n",
      "\n",
      "\n",
      "# Evaluation of Standard vs. Adaptive Retrieval\n",
      "\n",
      "## Query 1: What is Explainable AI (XAI)?\n",
      "*Query Type: Factual*\n",
      "\n",
      "**Reference Answer:**\n",
      "Explainable AI (XAI) aims to make AI systems transparent and understandable by providing clear explanations of how decisions are made. This helps users trust and effectively manage AI technologies.\n",
      "\n",
      "**Standard Retrieval Response:**\n",
      "Based on the provided context, Explainable AI (XAI) is a set of techniques aimed at making AI decisions more understandable, enabling users to assess their fairness and accuracy. The goal of XAI is to provide insights into how AI models make decisions, enhancing trust and accountability in AI systems. This involves developing methods for explaining AI decisions, which can help users understand the reliability and fairness of AI-driven outcomes.\n",
      "\n",
      "**Adaptive Retrieval Response:**\n",
      "Explainable AI (XAI) is a subfield of artificial intelligence (AI) that aims to make AI systems more transparent and understandable. The primary goal of XAI is to provide insights into how AI models make decisions, thereby enhancing trust and accountability in AI systems.\n",
      "\n",
      "XAI techniques are being developed to explain the reasoning behind AI decisions, making it possible for users to assess the reliability and fairness of AI outputs. This is particularly important in high-stakes applications such as medical diagnosis, finance, transportation, and manufacturing, where the consequences of AI errors can be severe.\n",
      "\n",
      "By making AI systems more explainable, XAI techniques can help address concerns about the potential for unintended consequences, accountability, and responsibility in AI development and deployment.\n",
      "\n",
      "**Comparison Analysis:**\n",
      "**Comparison of Standard Retrieval Response and Adaptive Retrieval Response**\n",
      "\n",
      "The two responses provide similar information about Explainable AI (XAI), but they differ in their tone, structure, and level of detail. Here's a detailed comparison of the two responses:\n",
      "\n",
      "**Accuracy and Relevance**\n",
      "\n",
      "* Both responses accurately convey the main idea of XAI, which is to make AI systems more transparent and understandable.\n",
      "* However, the Standard Retrieval Response provides more context and background information about XAI, including its goals and applications.\n",
      "* The Adaptive Retrieval Response is more concise and to the point, but it lacks the depth and detail of the Standard Retrieval Response.\n",
      "\n",
      "**Comprehensiveness**\n",
      "\n",
      "* The Standard Retrieval Response provides a more comprehensive overview of XAI, including its techniques, goals, and applications.\n",
      "* The Adaptive Retrieval Response focuses primarily on the definition and purpose of XAI, without providing much additional context or information.\n",
      "* The Standard Retrieval Response also highlights the importance of XAI in high-stakes applications, such as medical diagnosis, finance, transportation, and manufacturing.\n",
      "\n",
      "**Alignment with Reference Answer**\n",
      "\n",
      "* Both responses align with the reference answer, but the Standard Retrieval Response is more closely aligned due to its more detailed and comprehensive explanation of XAI.\n",
      "* The Adaptive Retrieval Response is more concise and to the point, but it may not fully capture the nuances and complexities of XAI.\n",
      "\n",
      "**Strengths and Weaknesses**\n",
      "\n",
      "**Standard Retrieval Response**\n",
      "\n",
      "Strengths:\n",
      "\n",
      "* Provides a more comprehensive overview of XAI\n",
      "* Offers more context and background information\n",
      "* Aligns closely with the reference answer\n",
      "\n",
      "Weaknesses:\n",
      "\n",
      "* May be too lengthy or wordy for some readers\n",
      "* Lacks the concise and to-the-point style of the Adaptive Retrieval Response\n",
      "\n",
      "**Adaptive Retrieval Response**\n",
      "\n",
      "Strengths:\n",
      "\n",
      "* Is concise and to the point\n",
      "* Provides a clear and direct definition of XAI\n",
      "* May be more suitable for readers who prefer a brief overview\n",
      "\n",
      "Weaknesses:\n",
      "\n",
      "* Lacks depth and detail\n",
      "* Fails to provide much additional context or information about XAI\n",
      "* May not fully capture the nuances and complexities of XAI\n",
      "\n",
      "**Conclusion**\n",
      "\n",
      "The Standard Retrieval Response provides a more comprehensive and detailed overview of XAI, while the Adaptive Retrieval Response is more concise and to the point. Both responses align with the reference answer, but the Standard Retrieval Response is more closely aligned due to its more detailed and comprehensive explanation of XAI.\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Run the evaluation comparing adaptive vs standard retrieval\n",
    "# This will process each query using both methods and compare the results\n",
    "evaluation_results = evaluate_adaptive_vs_standard(\n",
    "    pdf_path=pdf_path,                  # Source document for knowledge extraction\n",
    "    test_queries=test_queries,          # List of test queries to evaluate\n",
    "    reference_answers=reference_answers  # Optional ground truth for comparison\n",
    ")\n",
    "\n",
    "# The results will show a detailed comparison between standard retrieval and \n",
    "# adaptive retrieval performance across different query types, highlighting\n",
    "# where adaptive strategies provide improved outcomes\n",
    "print(evaluation_results[\"comparison\"])"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv-new-specific-rag",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
